Patient safety and rocket science.

To cite: McCulloch P. BMJ Qual Saf Published Online First: [please include Day Month Year] doi:10.1136/ bmjqs-2015-004863 ‘This safety stuff, it’s not rocket science’. Many readers of this journal will undoubtedly have heard this sentiment expressed by their clinical colleagues. The article by Kemper et al shows just how widely this impression of patient safety misses the mark. This high-quality study confirms the trend of the recent literature by finding that teamwork training using the civil aviation Crew Resource Management (CRM) approach has no evident clinical benefit, although it does seem to change attitudes and enhance some aspects of the ‘non-technical’ skills involved with interacting with colleagues. In doing so, the study highlights three areas of complexity and challenge in the development and evaluation of safety interventions. First, the interventions themselves are deceptively complex; as recommended by experts, they are grounded in theory, but may be entirely wrong. Second, the success of even ‘simple’ interventions like the WHO checklist hugely depends on the context and the implementation strategy. And third, the act of evaluation is far more difficult than it might first appear. Let us start from the end. By the methodological standards of safety and quality intervention studies generally, this is an exceptionally well done study. It is sizeable, involving six hospitals and over 8000 patients. There is a clear ‘PICO’ question as recommended by evidencebased medicine (EBM) pundits; the study protocol was published in advance; the study uses mixed methods intelligently to study outcomes in a structured way, using Kirkpatrick’s educational model; and there is even a control group. As a practitioner in the same field, the writer salutes the study group for their thorough and thoughtful approach. Yet, by the exacting standards of EBM, even this study would be regarded as being at moderate-to-high risk of bias. The allocation to groups is not random—intervention hospitals needed to sign up to certain financial and organisational standards and agreements, and their ability to do so may mean that they were in some way superior to the control hospitals. There is no attempt to blind the observers who evaluated communication, and whether the assessment of questionnaires was done in a blinded fashion is unclear. The large number of questionnaire-based instruments raises questions about their validity, reliability and independence of each other, not all of which the authors confront. Finally, there is no attempt to estimate the power of the study to detect the differences it sought, nor any explanation of what size of difference would be considered worthwhile. In contrast with many other such studies, Kemper and colleagues provide a very clear and detailed description of the CRM intervention, which has varied widely in previous studies in terms of training quantity, content and the percentage of staff receiving it. CRM was the ‘poster child’ of the patient safety movement before the WHO Surgical Safety Checklist usurped this role, and CRM remains a popular and (for some) profitable intervention because of the persuasive arguments for its effectiveness. These arguments boil down to: (a) it seemed to work for aviation; (b) it must make sense to make team members aware of how things can go wrong, and what good teamwork looks like; (c) it appeals (sometimes at a rather superficial level) to the findings of important psychological work on memory, perception and decision making. However, just because things ought to work does not prove that they do, and sometimes the theoretical basis for proposals for change is dangerously naïve. The implicit expectation in CRM studies is that the training will change the work culture in a beneficial way. Like the Francis report on egregious failings in care at Mid Staffordshire Hospital, which recommended ‘culture change’ to improve safety in the NHS, this ignores the stark reality that changing organisational culture is a massive task. Numerous examples in health systems, EDITORIAL